Children’s Literacy 
Initiative’s Blueprint 
for Early Literacy: 
Year 2 Evaluation Report 


Prepared by Research for Action * November 2018 


4 
Alyn McCarty, Ph.D. - Rachel Comly REA RESEARCH 
wy 


Kendra Strouf - Matthew Rigsby 


for ACTION 


About Research for Action 


Research for Action (RFA) is a Philadelphia-based nonprofit organization. We seek to use research 
as the basis for the improvement of educational opportunities and outcomes for traditionally 
underserved students. Our work is designed to strengthen public schools and postsecondary 
institutions; provide research-based recommendations to policymakers, practitioners, and the 
public at the local, state, and national levels; and enrich the civic and community dialogue about 
public education. For more information, please visit our website at www.researchforaction.org. 


Acknowledgments 


The Children’s Literacy Initiative (CLI) staff provided crucial support in setting up fieldwork and 
negotiating entry to research sites. They also served as helpful thought partners in negotiating 
challenges and questions that arose during the research. We would like to thank Brooke Di Leone, 
Research and Evaluation Manager; Kaitlin Lucas, Research Data Analyst; Luke Pasko, Research 
Assistant; and Jennifer Thomas, pre-K Supervisor. In addition, we appreciate CLI staff and coaches’ 
thoughtful participation in our interviews, which helped us gain a much richer understanding of the 
initiative. 


RFA would also like to acknowledge the support and cooperation of the participating center 
administrators and teachers. They have graciously welcomed us into their sites and taken time from 
their important work to help arrange our visits, send us data, and participate in surveys and/or 
interviews. 


This report was greatly enhanced by the contributions of many RFA staff. In addition to the authors, 
many RFA staff observed classrooms for the study, including Roger Chu, Jason Fontana, Dominika 
Krupocin, and Tosin Shenbanjo. Our intern, Clara Lee, contributed to multiple aspects of the report 
by supporting the research process with dedication and attention to detail. Our Communications 
Coordinator, Kathryn Carter, proofread this report. Katrina Morrison supported the development of 
qualitative research instruments, and Jill Pierce provided constructive feedback regarding 
qualitative analysis. We would also like to acknowledge the valuable oversight provided by Kate 
Shaw, Executive Director; Kate Callahan, Director of Quantitative Research; Tracey Hartmann, 
Director of Qualitative Research; Rebecca Reumann-Moore, former Director of Qualitative 
Research; and Alison Murawski, Chief Operating Officer and Director of Communications. 


We would also like to acknowledge Mary Eddins, Catherine Ellis, Rui Fu, Heidi Gross, Abdul-Qadir 
Islam, Allysen Marks, Nadia Mykysey, Katy Orlando, Brittany Russell, and Marilyn Tinari for 
conducting vocabulary assessments. 


The William Penn Foundation provided funding for implementation of Blueprint for Early Literacy 
and this evaluation research. Additional funding for Blueprint implementation was provided by The 
Pew Charitable Trusts, Boeing Company Charitable Trust, The 25 Century Foundation, The 
Caroline Alexander Buck Foundation, and The Capital Group Companies. Additional funding for the 
evaluation was provided by the Annie E. Casey Foundation. 


Table of Contents 


EXEC ULIVE SUNNY ices evecare ences arn cete eden ene eaicnaytenepeevaiv ep nese te rovad de anatase eerie ard ene ei i 
INtrOGUCU ON ira Mies Ge eisai dite hin itt Minis hiv Unpaid ad adenestla awd Galan ci bhiidiviaahs 1 
A. Summary of Evaluation And FindingS ..........c:eecesceeceesceeeeeeeeeeeseeeeeeaeeseeeaeeaeeaesaeesaeeaeeseaeaeeaeesaesaessaeeaeesaesaeeaeseaeeeeeaeseeseeneeenaess 2 
Bz OrgaMmiZationOf RO POM: ssecse-cevessevscvcesidebaceheceea Hist So eccccunecedengeces cuss Secgiaie ceed cca cheeeteves Sas cancsteegodeass dee stthose nev eeeetvevie av ete tara deste 2 
Study 1: Implementation of Blueprint in Philadelphia ..............ccsscccssecssesseesseecseecseeesseessseeeseessaeessueeseeeseessaeesaessaeessanersnenseeesseesaess 4 
A..Data'and Methodological ApprOac iva::ccisce.cecesseviascaicsscecedesdcacevsuncetesecaveccvtuaecdes cnaecasesasecassieveuaceceveubepietcadisagecebesaentunecsusdanenecvee 4 
B. Selection and Characteristics of Study Pre-K CenterS...........:cesccesesseceseeeeceeeeseeeeeeeeseeaeeaeesaeeaeeseeeaesseeeaeesaeeaesaeesaeeaeeeaeseeeeeeaes 6 
C. Blueprint Training and Coaching: .c.-ci-.c.cisecsseseeeeee setae vente wee pe cegeenteesevenceceetuenep eee dente eanepevendennvseeeaeise sea tnaieeiiy ed Siem secteenneemniens 8 
D. Implementing Blueprint in the ClaSSrOOM..........:cccsccsscssssssessseeeseeesseesstesseeesseecseueceeeeseeesrecseeseseesenessaueceneuseesesnessceseseeesanesaes 15 
E: Conclusions:and RECOMMeNdatiONS 's-:i.2vcrsi5.cceesecdi enters el ceed eels ine a lesen edie eel eee 20 
Study 2: Impact of Blueprint on Teachers ANd StUCeNts..........:cccscccssccsseecseeeseeecseeecseecseeesaeecsneesseesaeecsaeeceueesaeesanessaeeseeseseeeseeeeseeess 22 
A: Data‘and Methodological Approach riscccsiccccescccecescceesdcteecbeceseacitusvacceeestesctscccenscceedevoaseeudcteusiveussseveveses evdeqeceesneusaveleusvecctees 22 
B. Impact Of BIUEPrint ON TEACNESS.........c:cccesccescsseecseecseeesseecnecsneeeeecseeessueesaeesaeessneesaneceessaescsuessaeseseeseeeeeeeesaeesanessaessnnsesnesaes 23 
C.Impact:of BIUEprintOM Students wi: ccsianivlacccsecni antares ines ils aida eel eed aeeeend aa eevee 30 
CONCIUSIONS ANd: RECOMMENGALIONS cscs iies deepsea teenage telecasts eleeie adie nae e ese etal ieennavevene Gateteniiereden ieee 35 
Appendix A: Data Collection and Analytic Me@thods...........:ccssccsscssessseesseecseecseeesneesaeeeseesseeesaeeesuessanecseeesaeseaeessaneseaeesaeesanessaeeesneess 37 
A. Teacher Survey Sample De@SCIiption ...........c.:ccesscsssecsseesseeesseecssensseesseeeseuecsueeseeeseesseeeeseesseeesanessneseeeeseeseueeseeseeesseeseseteseeesees 37 
Be IMCS VG WS cece stares aces cerca sata cot cece cs nec ees nascar c cua acct an cea ces vaca ca sata ns a cee seach aay eg eeepc ec uchsnes eee ees esse cacti 37 
C: Baseline Equivalence ac:icaitiiawenitecd Bia tianekenin Aiveelaiiaiieled eiveiastiee Anahi etveneeteeng ered Wad ave ateenveveres 38 
De Obsérvations:of GlASSHOOMS i. ists tes eer nineteen tl ee anne ain kare ed 40 
E.Student Outcomes: Direct ASSESSMENT ae... scstessbketeesiee bar cetecied Av seaveutioeeecelcnveaniveeirnsieaenns cueede cueenccenaats 41 
FRETOREMICOS cst ste 8S teas els see Seca ee ae as ee ret A Cee eae tO es ei a ae teen 44 


Copyright © Research for Action, 2018 


am 


REA RESEARCH 
wy 


for ACTION 


Children’s Literacy Initiative’s Blueprint for Early 
Literacy: Year Two Evaluation Report 


Executive Summary 


Introduction 


Students who have not learned to read by the time they enter fourth grade are four times more 
likely to drop out of school, and this risk is even greater for low-income children.i In 2017, only 
35% of third grade students in Philadelphia reached this important milestone.i' The Children’s 
Literacy Initiative (CLI) aims to address this significant challenge by supporting children before 
they start kindergarten through its Blueprint for Early Literacy program. The Blueprint program 
combines literacy-based materials and curricula with professional development for pre-K teachers 
through workshops on best practices for early literacy instruction and individual coaching for 
educators. The Blueprint curriculum uses best practices in early childhood programs to strengthen 
language and literacy skills through play-based instruction. Table ES1 displays the resources, 
activities, and outcomes that comprise the Blueprint theory of action, used as a framework for this 
report. 


Table ES1. The Children’s Literacy Initiative’s Blueprint for Early Literacy theory of action 


RESOURCES/ ACTIVITIES OUTCOMES FOR OUTCOMES FOR 
MATERIALS TEACHERS STUDENTS 


e Blueprint for Early e Professional e Increased teacher e Increased student 
Literacy curriculum development through knowledge about early engagement in 
training and coaching literacy development language and literacy 
e High-quality children’s and instruction learning 
literature e Implementation of 
curriculum e Increased teacher e Increased student 
ability to implement mastery of early 
effective early literacy literacy concepts and 
instruction skills 


Increased teacher 
ability to create a 
positive classroom 
culture and a literacy- 
rich classroom 
environment 


Research for Action (RFA) is conducting a three-year external evaluation of this program. This 
report is comprised of two studies that provide in-depth findings of Year Two Blueprint 
implementation (resources and activities) and impacts (teacher and student outcomes). A synthesis 
of results across the two studies is provided below. 


Study 1: Blueprint Implementation 


Research for Action conducted a descriptive study of the quality of implementation of Blueprint in 
11 Philadelphia pre-K centers. We assessed the professional development CLI provides through 
training workshops on best practices for early literacy instruction and individual coaching for 
educators—key features of the Blueprint for Early Literacy program. This study also followed up on 
findings from Year One, including an in-depth exploration of challenges to implementation— 
consistent attendance at trainings, finding time for coaching conferences, and coaching amidst high 
teacher turnover—and CLI strategies to address them. 


Study 2: Impact of Blueprint on Teachers and Students 


To understand Blueprint’s impact on teacher and student outcomes, we conducted a rigorous 
causal impact study that employed a mixed-methods quasi-experimental research design, involving 
11 centers receiving Blueprint professional development and curriculum and 11 centers serving as 
a comparison group. 


Summary of Findings 


Blueprint implementation and impacts were strong despite evidence of limited training and 
coaching due to teacher turnover and variable attendance. 


This Year Two evaluation indicates that most lead teachers demonstrated high fidelity to the key 
elements of the Blueprint approach in the classroom (Message Time Plus, Intentional Read Aloud, 
and Power of Three), though issues related to differentiating instruction and the simultaneous 
implementation of Blueprint and Creative Curriculum persisted for some. There is strong evidence 
of impact on teachers and students: multiple data sources demonstrated that teachers and children 
in Blueprint centers benefitted from the Blueprint curriculum and professional development. 
Children in CLI-served classrooms made 2-3 months of additional progress in vocabulary 
development compared to children in similar classrooms not supported by CLI. Though less than a 
quarter of lead teachers in Blueprint classrooms in Spring 2018 received intended amount of 
training and coaching due to high turnover and variable attendance, most teachers had at least 
attended the Introduction to Blueprint 3.0 training and received at least one full year’s worth of 
coaching (over 20 hours). 


' Hernandez, Donald J. "Double Jeopardy: How Third-Grade Reading Skills and Poverty Influence High School Graduation." Annie E. Casey 
Foundation (2011). 

« School District of Philadelphia SY 2016-2017 District Scorecard." May 2018. 
https://cdn.philasd.org/offices/performance/Open_Data/School_Performance/District_Scorecard/DPR_SY1617_District_Scorecard_201 
80503.pdf. 


4 
RFA RESEARCH 


Children’s Literacy Initiative’s Blueprint for Early 
Literacy: Year Two Evaluation Report 


Prepared by Research for Action e November 2018 


Introduction 


Students who have not learned to read by the time they enter fourth grade are four times more 
likely to drop out of school, and this risk is even greater for low-income children.! In 2017, only 
35% of third grade students in Philadelphia reached this important milestone, as measured by the 
Pennsylvania State Standardized Assessment.2 


The Children’s Literacy Initiative (CLI) aims to address this significant challenge by supporting 
children before they start kindergarten through its Blueprint for Early Literacy program. The 
Blueprint program combines literacy-based materials and curricula with professional development 
for pre-K teachers through workshops on best practices for early literacy instruction and individual 
coaching for educators. Lead teachers are invited to participate in three core Blueprint trainings: 1) 
Introduction to Blueprint 3.0, 2) Message Time Plus, and 3) Intentional Read Aloud. Additionally, 
CLI pairs each lead teacher with a coach that provides 20 hours of in-person coaching annually. 


In addition to professional development through training and coaching, CLI provided teachers with 
pre-K literacy materials such as daily lesson plans, children’s books, and a structured and 
sequential theme guides. The Blueprint curriculum uses best practices in early childhood programs 
to strengthen language and literacy skills through play-based instruction. The implementation of 
the curriculum, which can either be used on its own or paired with a curriculum already in use, is 
structured around three key elements: 


e Message Time Plus (MTP) is a daily modeled writing and shared reading instructional 
practice. Teachers brainstorm, plan, and write text in front of students, then engage ina 
shared reading of the text. 


e Intentional Read Aloud (IRA) is a daily structured practice during which teachers model 
fluent reading and reading behaviors as teachers and students think about, talk about, and 
respond to text before, during, and after reading. 


+ Hernandez, Donald J. "Double Jeopardy: How Third-Grade Reading Skills and Poverty Influence High School Graduation." Annie E. Casey 
Foundation (2011). 

2 "School District of Philadelphia SY 2016-2017 District Scorecard." May 2018. 
https://cdn.philasd.org/offices/performance/Open_Data/School_Performance/District_Scorecard/DPR_SY1617_District_Scorecard_201 
80503.pdf. 


e Power of Three is a tool that shifts teachers away from focusing on classroom rules toward 
encouraging students and teachers to share classroom responsibilities. Using the Power of 
Three as a framework integrated into the culture of the classroom, teachers help students 
gain skills to “Take care of yourself, each other, and our classroom." 


A. Summary of Evaluation and Findings 


In January 2017, the William Penn Foundation provided CLI with funding to implement Blueprint in 
11 pre-K centers in Philadelphia for three years.3 CLI asked RFA to conduct a three-year 
independent, third-party evaluation of this work. Our Year One report, Children’s Literacy 
Initiative’s Blueprint for Early Literacy: Year One Evaluation, was a descriptive study of 
implementation in 2017 and provided CLI with formative feedback and data on early indicators of 
progress. 


Blueprint implementation and impacts were strong despite evidence of limited training and 
coaching due to teacher turnover and variable attendance. 


This Year Two evaluation indicates that most lead teachers demonstrated high fidelity to the key 
elements of the Blueprint approach in the classroom (Message Time Plus, Intentional Read Aloud, 
and Power of Three), though issues related to differentiating instruction and the simultaneous 
implementation of Blueprint and Creative Curriculum persisted for some. There is strong evidence 
of impact on teachers and students: multiple data sources demonstrated that teachers and children 
in Blueprint centers benefitted from the Blueprint curriculum and professional development. 
Children in CLI-served classrooms made 2-3 months of additional progress in vocabulary 
development compared to children in similar classrooms not supported by CLI. Though less than a 
quarter of lead teachers in Blueprint classrooms in Spring 2018 received intended amount of 
training and coaching due to high turnover and variable attendance, most teachers had at least 
attended the Introduction to Blueprint 3.0 training and received at least one full year’s worth of 
coaching (over 20 hours). 


B. Organization of Report 


The resources, activities, and outcomes that comprise the Blueprint theory of action provide a 
framework for this report, which examines Blueprint implementation and impact in academic year 
2017-18 (Table 1). 


3 In nine of the 11 treatment sites, one lead teacher was selected to receive training and coaching and begin implementing Blueprint in 
Fall 2016. This earlier initial implementation was funded by The Pew Charitable Trusts. 


Table 1. The Children’s Literacy Initiative’s Blueprint for Early Literacy theory of action 


IMPLEMENTATION STUDY IMPACT STUDY 
MATERIALS TEACHERS STUDENTS 
e Blueprint for Early e Professional e Increased teacher e Increased student 
Literacy curriculum development through knowledge about early engagement in 
training and coaching literacy development language and literacy 
e High-quality children’s and instruction learning 
literature e Implementation of 
curriculum e Increased teacher e Increased student 
ability to implement mastery of early 
effective early literacy literacy concepts and 
instruction skills 


Increased teacher 
ability to create a 
positive classroom 
culture and a literacy- 
rich classroom 
environment 


This report is comprised of two studies that provide in-depth findings of Year Two Blueprint 
implementation (resources and activities) and impacts (teacher and student outcomes). 


e Study 1: Implementation of Blueprint in Philadelphia. We conducted a descriptive study 
of the quality of implementation of Blueprint in 11 Philadelphia pre-K centers, focusing on the 
extent and quality of teacher professional development and curriculum implementation. This 
study also follows up on findings from Year One, including an in-depth exploration of 
challenges to implementation and CLI strategies to address challenges. This study draws on 
multiple methods of data collection and analysis, including administrative data on training 
and coaching, surveys of teachers, interviews with CLI staff and coaches, and interviews with 
center directors and teachers. 


e Study 2: Impact of Blueprint for Early Literacy. To understand the impact of Blueprint on 
teacher and student outcomes, we conducted a rigorous causal impact study that employed a 
mixed-methods quasi-experimental research design involving 11 centers receiving Blueprint 
professional development and curriculum and 11 centers serving as a comparison group. This 
study draws on cross-verification of multiple methods of data collection and analysis: direct 
assessments of student literacy skills; researcher observations of the language and literacy 
environment of classrooms; teacher surveys; and interviews with key CLI staff, coaches, and a 
sample of directors and teachers in centers receiving Blueprint. 


Our report ends with a review of key findings from the implementation and impact studies and 
suggestions for implementation in Year Three, the final year of the intervention. 


Study 1: Implementation of Blueprint in Philadelphia 


This study evaluates the quality of Blueprint implementation in Philadelphia, including training, 
coaching, and implementation of the Blueprint curriculum in the classroom. Additionally, following 
up on our research in the Year One implementation report, we examined CLI strategies for 
addressing several significant implementation challenges. This report section is organized into six 
sub-sections: 


e Data and Methodological Approach. In this section, we discuss guiding research 
questions and data collection activities that served as the foundation of the implementation 
study. 


e Selection and Characteristics of Study Centers. CLI selected 11 pre-K centers to receive 
Blueprint professional development and curriculum during the study period. In this section, 
we discuss the selection criteria and center characteristics of selected sites, including a 
spotlight on teacher turnover. 


e Fidelity of Blueprint Professional Development. This section examines the extent and 
quality of Blueprint training and coaching, including in-depth descriptions of challenges for 
training and coaching—inconsistent attendance, supporting productive coach-teacher 
conferences, and coaching in the context of high teacher turnover—and CLI strategies for 
addressing them. 


e Quality of Blueprint Implementation in the Classroom. This section examines Blueprint 
curriculum implementation in the classroom, focusing on teacher fidelity of implementation 
of key Blueprint components: Message Time Plus, Intentional Read Aloud, and Power of 
Three. We also provide in-depth descriptions of two implementation challenges— 
differentiation of instruction and using Blueprint with Creative Curriculum—and CLI 
strategies for addressing them. 


e Conclusion and Recommendations. We summarize our main findings and offer 
recommendations for Blueprint implementation in Year Three. 


A. Data and Methodological Approach 


In this section, we describe the research questions and data collection activities that guided the 
implementation study. 


Research Questions 
In Year Two, RFA’s implementation study specifically addressed four questions: 
1. What is the quality of implementation of the Blueprint program? 


2. What percentage of lead teachers received the full suite of training and coaching supports 
from CLI? 


3. Howhas CLI addressed the challenges identified in the first year of the project for training, 
coaching, and implementing the Blueprint curriculum? 


4. What additional strategies, if any, do teachers and directors use to successfully address 
challenges to Blueprint professional development and curriculum implementation? 


Data Collection Activities 


This descriptive implementation study draws on triangulated data from the following data 
collection activities (see Appendix A for more information): 


e CLI administrative data on training and coaching hours for lead teachers. CLI provided 
data on participation in Blueprint training and coaching for lead teachers (N=34) in Year 
Two of the study.#5 


e End-of-year survey of lead pre-K teachers. In Spring 2018, RFA developed and 
administered an online survey of lead teachers in Blueprint centers, including their 
perceptions of Blueprint training, coaching, and curriculum. Eighty percent of consenting 
lead teachers responded to the survey (N=25). 


e Interviews with CLI staff and coaches. RFA interviewed relevant CLI staff and all four CLI 
Blueprint coaches in Spring 2018 to understand CLI strategies for addressing Year One 
challenges. 


e Interviews with a sample of center directors and lead teachers at four centers 
receiving CLI support. To go deeper into implementation challenges raised in our Year 
One report and to understand director and teacher perspectives on CLI strategies for 
addressing Year One challenges, RFA interviewed a purposeful sample of center directors 
and teachers at four treatment sites. Specifically, we sampled sites with higher and lower 
rates of teacher turnover, and we sampled teachers according to their level of experience 
with Blueprint. 


e Observations of Blueprint professional development meetings. In Years One and Two, 
RFA observed three director meetings, one teacher meeting, and four Blueprint trainings. 
These observations provided background information to support the analysis of Blueprint 
implementation. 


e Document review. RFA reviewed Blueprint curriculum materials, the CLI Blueprint 
website, and training materials to support the analysis of Blueprint implementation. 


4 CLI coached and trained teachers in 35 classrooms. However, one lead teacher opted out of the study. The data presented here excludes 
this classroom. 

5 We define “lead” teacher as the CLI teacher targeted to participate in trainings and receive coaching. In some cases, this teacher was not 
officially a lead teacher but an assistant acting as lead because of staffing challenges. In other cases, there were technically two lead 
teachers in the classroom. However, CLI typically only coached one teacher per class. 


B. Selection and Characteristics of Study Pre-K Centers 


Selection of Centers for the Study 
CLI used the following criteria to select 11 Philadelphia centers for participation in this study: 


e High need. CLI selected centers in low-income areas with a high need for quality childcare. 
According to recent census data, we found that study centers are in neighborhoods where 
about three in five children live in households with single parents and a third of families are 
living in poverty.® 


e Geography. CLI wanted to grow its work in areas where the initiative already hada 
presence and expand into new high-need areas. Study sites are located in North, South, and 
West Philadelphia. 


e Quality ratings. Centers needed to have at least a Star 3 rating in the Keystone STARS 
system, an initiative of the PA Office of Child Development and Early Learning (OCDEL). 
Stars 3 and 4, the highest rating on a 1-4 scale, indicate strong center quality. 


e Expand within networks. CLI already had a relationship with one large network of 
childcare centers. For the current study, they expanded to some additional sites within this 
network. Other sites include those affiliated with the School District of Philadelphia and 
non-network sites providing center-based pre-K. 


e Director buy-in. Through director interviews, CLI sought to identify sites where the leader 
would support Blueprint. 


e Centers willing to adopt a new or additional curriculum. Some centers which had just 
adopted a new literacy curriculum could not participate. Centers which had existing 
curricula needed to be open to adding Blueprint. Most sites were using Creative Curriculum. 


Staffing and Student Enrollment Challenges 


As is typical in pre-K environments, over-time shifts in student enrollment and teacher turnover 
resulted in an evolving target for CLI during their implementation of the Blueprint program. By the 
end of Year One in June 2017, there were 37 classrooms receiving supports from CLI, each with one 
lead teacher identified by CLI for coaching and training. By the end of Year Two in Spring 2018, only 
19 of these lead teachers remained in their positions. These significant staffing changes have 
considerable implications for CLI’s professional development approach, as discussed below. 


6 United States Census Bureau. (2017). 2017 TIGER/Line Shapefiles [Data file]. Retrieved from https://www.census.gov/geo/maps- 
data/data/tiger-line.html 


~ 
Ww Spotlight on Teacher Turnover 


The pre-K centers in this study experienced teacher turnover at rates higher than 
national estimates. By the end of Year One, there were 37 lead teachers across the 11 
centers receiving Blueprint training, coaching, and curriculum materials. By the end of 
Year Two, only 19 of these teachers remained in their positions, a teacher turnover rate 
of 48%. Annual pre-K teacher turnover is estimated at 30% nationally.” In interviews, 
RFA asked teachers what is needed to support pre-K teacher retention. 


Teacher feedback regarding retention: 


e Compensation should support the cost of living. One teacher from a site with low 
turnover explained that a key reason that teachers continue to work in or leave pre- 
K sites is how much they are paid: “No matter how much you like your job, you still 
have to live.” Another teacher, also from a site with low turnover, added that 
benefits are key factors in determining whether to stay at a site or leave: “I came 
from [another] preschool, and I couldn't stay... 1 didn't have good benefits, I was 
paying a lot of money for my benefits.” 


high turnover explained, “I think a lot of teachers feel like they don't have the 
support from management that they need.” Other teachers from sites with lower 
turnover rates pointed to support from the center administration or their colleagues 
as essential. One of these teachers highlighted the administration’s support 
“communicating with parents [and] helping with the material that’s needed in the 
classroom” as important, while another teacher identified a “wonderful” teaching 
partner as a key reason for continuing to work at her site for many years. 


e Professional growth should be supported. Multiple teachers who worked in 
centers where they received support for their professional growth named this 
support as key to retention. One teacher reported that teachers want to continue 
working at sites that are “challenging you to grow and providing you with the tools.” 
Another teacher agreed, explaining that their center director encouraged them to get 
a bachelor’s degree, which enabled them to become a lead Head Start teacher. 


e Teachers should feel valued. One school district teacher described how critical it is 
to be valued for their work: “You know [as pre-K teachers,] we're still doing as much 
work as the grade teachers, if not more, because the students are very needy, and 
they're just learning these things for the first time. So, it's really important that 
you're valued on the same level as the grade teachers.” 


I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
\ e Teachers’ work in the classroom should be supported. A teacher from a site with 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 


7 Preschool Curriculum Evaluation Research Consortium. "Effects of preschool curriculum programs on school readiness: Report from the 
Preschool Curriculum Evaluation Initiative." Washington, DC: National Center for Education Research, Institute of Education Sciences, US 
Department of Education. 2008. 


C. Blueprint Training and Coaching 


In this section, we assess the professional development CLI has provided to lead pre-K teachers so 
far through training workshops on best practices for early literacy instruction and individual 
coaching for educators—key features of the Blueprint for Early Literacy program. In addition, our 
research examined three challenges for training and coaching identified in Year One: consistent 
attendance at trainings, finding time for coaching conferences, and coaching amidst high teacher 
turnover. For the Year Two evaluation, RFA continued to track these challenges and CLI’s efforts to 
address them. 


Key Findings: Blueprint Training and Coaching 


CLI strategies to address implementation challenges: 


CLI offered annual teacher and director collaboration meetings and multiple 
opportunities for lead teachers to attend core Blueprint content trainings, 
employing various strategies to encourage attendance. 


CLI provided training for coaches to support productive in-the-classroom coach- 
teacher conferences. 


Implementation successes: 


Teachers new to centers reported that trainings laid the foundation for smoother 
implementation of Blueprint in the classroom. 


Teachers new to centers reported that Blueprint coaching built their “teaching 
toolbox.” 


Implementation challenges: 


Forty-eight percent of lead teachers were new to their centers from the end of 
Year 1 to the end of Year 2. 


Very few centers and classrooms were exposed to the full suite of CLI professional 
development activities, due to high turnover and variable attendance. 


CLI did not provide explicit guidance to coaches about coaching in the context of 
high turnover. 


It remains logistically challenging for coaches and teachers to find the time to 
conference during their coaching sessions while class is in session. 


Most respondents agree that in-the-classroom coach-teacher conferences are not 
ideal. 


Blueprint Training 


From January 2017 to June 2018, CLI invited all lead teachers in centers receiving Blueprint to 
participate in three core Blueprint trainings: 1) Introduction to Blueprint 3.0, 2) Message Time Plus, 
and 3) Intentional Read Aloud.® Teachers received $100 stipends for attending each training event, 
and make-up trainings were offered for teachers who missed scheduled trainings. 


CLI employed several strategies to encourage training attendance. 


In Year One, we reported that about 70% of lead teachers attended both trainings offered 
(Blueprint 3.0 and Message Time Plus).? CLI aimed to improve attendance rates in Year Two using 
the following strategies: 


e Strategic scheduling. CLI scheduled trainings on weekends because, as noted by a CLI staff 
member, it is difficult for centers to find coverage for classes during the work week. In 
addition, CLI reported strategizing with directors about scheduling. Still, teachers and 
directors cited the date and time of trainings as the biggest impediment to attendance, 
reporting that what kept them from attending weekend trainings and meetings included 
scheduling conflicts and the need to recuperate after a long week. 


e Motivational facilitators. A CLI staff person reported strategically selecting experts in the 
pre-K field with whom pre-K teachers feel comfortable “so people are more motivated to 
come and attend trainings.” 


e Compensation for time. As stated above, CLI compensates teachers for attending trainings. 
One teacher said, “It's great that they do their paid trainings. That's a great incentive as well. 
That's really nice to feel valued for your time.” 


Only a third of lead teachers attended all three core Blueprint trainings, due to high 
teacher turnover and low attendance. 


We used data from CLI’s administrative records to assess how many lead teachers attended the 
three core CLI trainings by the end of Year Two. As Figure 1 demonstrates, of the 34 lead teachers 
working in Blueprint centers in June 2018,!° about three in four had attended an Introduction to 
Blueprint 3.0 training, half had attended a Message Time Plus training, and two in five attended an 
Intentional Read Aloud training. About a third of lead teachers attended all three trainings. CLI also 
encouraged, but did not require, assistant teachers to attend trainings. 


8 In nine of the 11 treatment sites, one lead teacher was selected to receive training and coaching and begin implementing Blueprint in 
Fall 2016. This initial implementation was funded by The Pew Charitable Trusts. 

° Our Year One report excluded two Blueprint centers because of delayed research approval, so these data are based on 9 treatment 
centers. 

10 CLI coached and trained teachers in 35 classrooms in Year Two. However, one lead teacher opted out of the study. The data presented 
here excludes this classroom. 


Figure 1. Percent of lead Blueprint teachers who attended core CLI trainings by June 2018 
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Source: CLI administrative data, 2016-17 and 2017-18 
Note: Retained lead teachers are those that have been working as leads in centers since Spring 2017. 


Though most interviewed teachers and directors reported that it is difficult to find time to 
participate in trainings and meetings, we found that another factor driving low training rates was 
the high rate of teacher turnover. As noted above, of the 34 lead teachers working in Blueprint 
centers in June 2018, only 19 were returning teachers from Year One. Figure 1 shows that training 
attendance was higher for this group, with over 40% of retained teachers having attended all three 
trainings. 


Blueprint Coaching 


CLI pairs each lead teacher with a coach that provides 20 hours of in-person coaching annually. In 
some cases, assistant teachers also receive coaching, though this was not done systematically 
across centers. An ideal Blueprint coaching session involves two conversations between the coach 
and teacher (a “pre-conference” and a “post-conference”) before and after the coach models a 
lesson or the coach and teacher implement a lesson together (“co-teach”). In Years One and Two, 
coaching goals included supporting teachers in 1) consistently implementing Blueprint and 2) 
developing a strong literacy environment and classroom culture. In Year Two, CLI also included 
support for teaching writing and addressing specific challenges related to implementing Blueprint 
in the classroom, particularly differentiating Blueprint for younger children and English Language 
Learners and implementing Blueprint with Creative Curriculum. 


10 


Because of high turnover, only two in five lead teachers received the full amount of 
CLI coaching. 


We used data from CLI’s administrative records to assess the number of hours of coaching received 
by lead teachers in CLI-supported classrooms by the end of Year Two. Figure 2 shows that, in Year 
Two, only 38% of lead teachers had received at least 40 hours of coaching. 


Figure 2. Percent of lead Blueprint teachers by hours of coaching received by June 2018 


100% 
90% 
80% 
4 68% 
2 70% 
© 
o 
2 60% 
& 50% 
= 
oO 9 
= 40% 35% ooh 
3 32% 
“5 30% 26% 
x 
20% 
10% 
0% 
0% 
Less than 20 20-39 hours 40 or more 
hours of coaching of coaching hours of coaching 
= All Lead Blueprint Teachers (N=34) @ Only Retained Lead Blueprint Teachers (N=19) 


Source: CLI administrative data, 2016-17 and 2017-18 
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As with the low training rates discussed above, the high rate of teacher turnover limited the extent 
to which teachers working in centers in Spring 2018 had received the intended 40 hours of 
coaching. Among teachers who stayed in their study site positions from June 2017 to June 2018, 
68% received at least 40 hours of coaching. 


Coaches and CLI staff report that coaching in the context of high teacher turnover 
continues to be a challenge. 


In Year One, we found that coaching within the context of high teacher turnover was a major 
challenge for CLI. Our findings from Year Two suggest that turnover continues to be a challenge for 
productive coaching. In interviews with CLI staff and coaches, we probed deeper into this challenge 
and found that it involves two related problems: 1) It is difficult to productively coach new teachers 
before they have received any training, and 2) CLI must strategize and reallocate coaching hours to 
coach new teachers. 
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Coaching new teachers prior to trainings. Coaching new teachers before they have attended 
trainings is a challenge, as all four coaches reported. One coach described new teachers’ 
understanding of Blueprint as “kind of foggy until they attended the training.” This coach continued 
to explain the importance of attending trainings to fully benefit from coaching: 


When the teachers go for training, it's usually a five-hour training. When the coaches 
come into the classroom, we can stay no longer than between an hour to three hours, 
so we can't do everything in an hour, so it's really imperative that they attend the five- 
hour training. If they actively attend the five-hour training, a light bulb goes on to say, 
"Oh, this is what [my coach] meant.” 


Allocating coaching hours. Two coaches and one CLI staff member reported that when they invest 
coaching hours in teachers who leave, they have to strategize and reallocate hours from the 
coaching bank to support new and interim teachers. A coach described the complicated challenge: 


Let's say I'll start coaching a teacher, and get maybe six hours, seven hours in. Feel like 
I'm making a little bit of progress. Then, that person leaves. I have to coach them for, I 
don't know how many, 20 hours or something. So, we have to decide, do I want to 
continue? Should I continue finishing up those hours with this brand-new teacher? 
That's number one. Number two, sometimes there isn't a new lead teacher in the room. 
The assistant teacher takes over. Should I coach them? Or, should I just put coaching 
on hold until they get a new lead teacher in? It's the logistics of the hours of coaching, 
that's one problem. 


A CLI staff person said that it is easier to reallocate coaching hours in the budget early in the year, 
but it becomes more difficult as the year goes on. 


Finding adequate time for coach/teacher conferences continued to be a common 
challenge across centers. 


In Year One, we reported that one of the primary challenges to productive coaching was finding 
time for coach-teacher conferences during coaching sessions. In Year Two, directors made more of 
an effort to ensure that teachers had time for coaching conferences, according to some coaches. 
However, overall, coaches and teachers continued to find it logistically challenging to find the time 
to conference during their coaching sessions. 


CLI trained coaches to support in-the-classroom conferences, but CLI staff, coaches, 
and teachers agree these are not ideal. 


Because of limited or no dedicated planning time for teachers in pre-K centers, and because centers 
need to maintain staff-child ratios to meet regulatory requirements, CLI trained coaches to help 
teachers build routines that support in-the-classroom conferences during teaching hours. For 
example, coaches learned to build routines to help children understand how to respect when their 
teacher is learning. Yet CLI staff, coaches, and teachers agree that in-the-classroom conferences are 
not ideal, as teachers must balance giving attention to the conference and to their class. For 
example, one coach explained that meeting in the classroom often makes it difficult “to be able to 
listen, and comprehend, and reflect on their practices, because [teachers] have one eye on the kids, 
and one eye on me.” 
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Summary of CLI Supports Received 


CLI aimed to provide the full suite of their supports for lead teachers in 11 Philadelphia pre-k 
centers by the end of Year 2 of implementation, including three core trainings and 40 hours of in- 
person coaching. As shown in Figure 3 below, only about one in five leads teaching in Blueprint 
classrooms in Spring 2018 received intended amount of training and coaching. However, a 
majority—nearly 60%—had at least attended the Introduction to Blueprint 3.0 training and 
received at least one full year’s worth of coaching (over 20 hours). 


Figure 3. Percent of lead Blueprint teachers by level of CLI supports received by June 2018 
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Source: CLI administrative data, 2016-17 and 2017-18 
Note: Retained lead teachers are those that have been working as leads in centers since Spring 2017. 


The figure also reports CLI supports for retained teachers, i.e., teachers that have been a part of the 
study since the beginning in Spring 2017. These data show that it is difficult to deliver the full suite 
of services even when teacher do not turn over. Less than 40% of retained teachers had received 
the full suite of CLI supports by the end of Year 2. However, nearly 80% of retained teachers 
attended at least the introductory Blueprint training and received at least 20 hours of coaching. 
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Ww Spotlight on Training and Coaching New Teachers 


Teacher turnover in study sites was higher than national averages, as reported above. 
Here, we highlight specific findings relevant to new teacher training and coaching. 


New teachers reported that trainings laid the foundation for smoother 
implementation. One teacher, new to implementing Blueprint in Year Two, described 
how the trainings supported Blueprint implementation: “It was very helpful because it 
actually broke more of the curriculum down for me...it actually gave me a lot of hints 
and tips that I can utilize in the classroom.” Similarly, another new teacher 
acknowledged the key role training played in becoming familiar with the curriculum: 
“After I did the trainings, after I learned it and did it multiple times, I felt like I got it 
down pat.” 


New Blueprint implementers reported that coaching built their “teaching toolbox.” One 
said that coaching bolstered her confidence with teaching and implementing Blueprint. She 
explained that coaching “makes me really, really, really comfortable with teaching and 
implementing [Blueprint], and you feel confident...that coaching really helped me to get 
comfortable with [Blueprint].” Another said that she appreciated the way coaches provided 
scaffolding “so it wasn’t overwhelming” to learn new practices. 


Some centers created structures to support new teachers with implementing 
Blueprint. CLI recognized the challenge of high turnover, but interviews suggest that 
there were no formal plans address this challenge. However, teachers and directors at 
two centers, one with higher turnover rates and one with lower turnover rates, 
described these strategies for supporting new teachers: 


e Designating a teacher to support and coach new teachers. One teacher, 
referred to by the director as their designated “internal coach,” reported 
supporting new teachers by using the CLI pre-K lesson plan templates to prepare 
them before implementing a lesson. The director said, “That’s just something we 
did internally. Not that it erased the need for a [CLI] coach, because I think that’s 
huge. That’s just how we kind of survived, if you will.” At another center, one 
teacher reported letting new teachers observe her and ask questions. She said “I 
have no problem with [new teachers] coming to me asking questions. I have no 
problems letting them sit in on Message Time Plus or whatever we're doing.” 


e Ensuring assistant teachers are involved in training and coaching. One CLI 
staff person reported training and coaching the assistant teacher at a center with 
high teacher turnover to avoid “losing the flow of what we’re doing there.” A 
teacher who was new to the lead role said that because of the opportunity to be 
involved in coaching as an assistant, “It wasn't like I was just all lost” when the 
lead teacher left the center. One director added that training assistants would also 
help empower them to develop their skills: “Even though our assistant teacher is 
the assistant, that does not mean that they can't support and take a piece of the 
curriculum implementation. It would be great if they had questions, or if they were 
implementing a portion of the curriculum, that they get feedback as well.” 


vee eee i ld 


14 


D. Implementing Blueprint in the Classroom 


In this section, we examine the implementation of Blueprint in the classroom in Year Two. We also 
follow up on identified Year One challenges: 1) differentiating Blueprint for younger children and 
English Language Learners and 2) implementing Blueprint with Creative Curriculum. 


Key Findings: Implementing Blueprint in the Classroom 


CLI strategies: 


e Through trainings for coaches and teachers, coaching activities, and resources, CLI 
has provided some support to mediate key challenges of implementing Blueprint. 


Successes: 


e Most surveyed teachers reported implementing key elements of Blueprint with 
fidelity. 
Challenges: 
e While differentiating Blueprint and implementing Blueprint with Creative 


Curriculum did not widely impact implementation across classrooms, these issues 
continued to be a challenge for some teachers. 


Implementation of Key Elements of Blueprint 


In addition to professional development through training and coaching, CLI provided teachers with 
the Blueprint curriculum, as described above. CLI recommends that teachers 1) implement Message 
Time Plus every day; 2) engage in Intentional Read Aloud every day; and 3) integrate the Power of 
Three into the culture of the classroom. We surveyed lead teachers in Blueprint classrooms to 
understand the frequency of implementation of these key elements. 


Most surveyed teachers report implementing key elements of Blueprint with fidelity. 


Figure 4 demonstrates that nearly all surveyed teachers implemented Message Time Plus and 
Intentional Read Aloud with fidelity, and most implemented Power of Three with fidelity. 
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Figure 4. Percent of surveyed lead Blueprint teachers implementing key elements of Blueprint with fidelity, 
N=25 teachers 
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Source: RFA teacher survey, 2017-18 
Note: Sample sizes vary by item from 24-26 teachers 


cette ete ia e tee ie I lel elle el ele el eel lol 


e 


AN Maintaining Implementation Fidelity: Spotlight on Assistant Teachers 
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In Year One, we found that variation in implementation fidelity was associated with I 
teacher buy-in and turnover and director support. In Year Two, coaches and staff I 
reported that, in addition to these factors, supportive assistant teachers promoted I 
strong Blueprint implementation. | 
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One coach explained that if teachers have a “good relationship with the assistant, and if 
that assistant is totally on board, that assistant can be a great resource to that teacher.” 
This coach said that assistants can sit on the rug with children during Intentional Read 
Aloud and Message Time Plus, “When that assistant is sitting on the rug with the 
students, it means that we are all working together. We're all on board here.” 


ee ee | 


Challenges for Implementing Blueprint in the Classroom 


In Year One, RFA identified two challenges related to implementing Blueprint in the classroom: 1) 
differentiating instruction to meet the needs of children at all skill levels and 2) implementing 
Blueprint with Creative Curriculum. In Year Two, RFA further investigated these challenges and the 
strategies CLI used to mitigate them. We surveyed lead teachers in CLI-supported classrooms and 
conducted interviews with CLI staff and a sample of directors and teachers with a range of support 
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and familiarity with CLI practices. Below, we provide an update on CLI’s progress in these areas 
during Year Two. 


Challenge #1: Differentiating instruction for children with different learning needs 


In Year One, some teachers identified challenges differentiating for younger or less-skilled students 
and did not have a shared understanding of whether it was permissible or desirable to adapt 
Blueprint to meet students’ needs. To address this challenge, CLI trained coaches and teachers in 
strategies to adapt Blueprint to meet the different needs of children. One coach described this as 
learning how to “break down a lesson and adapt it to the needs of the classroom.” In interviews, CLI 
coaches reported providing teachers with varied differentiation strategies: 


e Creating specific differentiated goals. One coach said, “We tell them to set specific goals 
for children when they're conferring with each child.” These goals can help teachers plan 
how to support each child’s learning. The coach provided an example: “Maybe if a child is 
learning the beginning sounds of letters, we'll certainly provide...letters with pictures for 
children who need to rely on pictures to understand letters.” 


e Using classroom staff strategically. For classrooms with assistant teachers, one coach 
recommended asking the assistant teacher to work with three-year-olds while the lead 
teacher works with older children. 


e Adapting seating arrangements. One coach said that they encourage English Language 
Learners or younger students to “sit close to the teacher.” 


e Providing small-group work. Coaches encouraged teachers to create a small group of 
students who need more instruction and “reteach” them lessons in a more targeted way. 


e Making learning more interactive. Coaches listed interactive activities that could be used 
to engage children in the learning process, including “turn-and-talks,” songs, and using 
more pictures. 


e Checking in on children’s understanding. One coach explained, “When they're reading a 
story, they should stop at least three times during the course of reading that story and ask 
open-ended questions or questions [relevant] to the story for comprehension to make 
certain that they do understand. You can't go so fast when you're working with children 
with different abilities.” 


e Incorporating children’s home languages. A coach described one way of incorporating 
home languages, explaining that if there is a story that is in English and Spanish, “the 
teacher will read it in English and in Spanish and then she will create a follow-up activity 
that uses the Spanish words and English words.” 


In interviews, teachers also reported using Message Time Plus scaffolds, substituting books or 
spending more time on them, shifting lesson objectives, using Blueprint’s tips for differentiating for 
English Language Learners, and connecting with families for extra support. Although differentiation 
strategies varied, teachers had a shared understanding that Blueprint lessons can be adapted to 
meet the needs of the children in their classrooms. 
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Adapting Blueprint for younger children and English Language Learners remains 
challenging. 


Our survey evidence suggests that some teachers continue to struggle to meet the needs of younger 
students and English Language Learners. As Figure 5 shows, more than a third (34%) of teachers 
reported that it was not easy to adapt Blueprint to meet the needs of younger students, and over 
half (52%) reported similarly for English Language Learners. In contrast, only 11% of teachers 
reported it was not easy to adapt Blueprint for children of different skill levels. 


Figure 5. Percent of lead teachers reporting ease of adaptability of Blueprint for children with different 
needs, N=25 teachers 
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Source: RFA teacher survey, 2017-18 
Note: Sample sizes vary by item from 24-26 teachers 


Challenge #2: Using Blueprint with Creative Curriculum 


Teachers at most Blueprint sites in our study use Creative Curriculum in addition to Blueprint. In 
Year One, negotiating the use of multiple curricula was the most frequently mentioned 
implementation challenge across all types of interviewees (CLI staff, coaches, administrators, and 
teachers). In some cases, teacher adaptations to accommodate both curricula resulted in lower- 
quality Blueprint implementation. To address this challenge, CLI staff interviewed teachers that 
were able to balance the two curricula to identify promising strategies for aligning Creative 
Curriculum studies with Blueprint themes. CLI created a resource document describing the overlap 
and divergence of the two curricula and provided lesson plan templates to assist teachers in 
strategizing about how to implement both. CLI held trainings for coaches and meetings for directors 
and teachers to share this content and discuss promising strategies. 


Challenges regarding the implementation of both Creative Curriculum and Blueprint 
persist for CLI coaches and some teachers. 


We asked teachers who use Blueprint and Creative Curriculum to tell us how easy or difficult it is to 


meet the requirements of Creative Curriculum while also using Blueprint (N=16 surveyed 
teachers). Responses were mixed, as shown in Figure 6. 
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Figure 6. Percent of surveyed lead teachers reporting ease of meeting requirements of Creative Curriculum 
while using Blueprint, N=16 teachers 
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Source: RFA teacher survey, 2017-18 


CLI staff acknowledged that, even with the strategies they have provided, finding ways to 
implement both curricula is “a lot to ask teachers to do.” One CLI coach said that helping teachers 
implement Blueprint with fidelity while satisfying the requirement to implement Creative 
Curriculum has been the “number one challenge” for her coaching. Teacher interviews also 
suggested the curriculum conflict continued to impede Blueprint implementation in Year Two for 
some. For example, one teacher reported that her coach told her to prioritize Intentional Read 
Aloud rather than other Blueprint components, given her reported time constraints. 


Although CLI provided lesson plan templates and a resource matching Blueprint themes to Creative 
Curriculum studies, some teachers and directors reported that it is still a challenge to fit it all in. Of 

matching themes to studies, one teacher said, “I tried it and it did not work. It's possible, but I think 
it's a little too much for the children to really grasp.” Another teacher explained: 


When it matches up, it's perfect, it runs smoothly. But when it doesn't, it can be a little, 

just a little tricky and overwhelming for the kids. Not for us so much because we know 
both curriculums and we have a plan for both, but just like, okay, we're learning about 
fairy tales and "reduce, reuse, recycle." So, it’s kind of like they need more of a scripted, 
one basic theme. So that’s a little challenging when they don't line up. 


In addition, two directors expressed concern for children’s development in domains other than 
literacy (for example, math, science, and social studies) when making decisions about what parts of 
Creative Curriculum and Blueprint to implement. 


One of these directors noted confusion about how to implement Blueprint with Creative Curriculum 
after the CLI director meeting focused on this challenge. This director left the meeting wishing they 
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had spent more time doing the work of planning to balance the curricula, so she could feel prepared 
to support teachers to use both: 


I got a little confused, and I think maybe it was a shortness of time that was expended on 
actually showing how to merge it together, as opposed to going over the historical 
background, and the individual parts. I think more time could have been spent on actually 
taking the theme [compared] to the theme of Blueprint or taking a study compared to the 
theme, and how we integrated throughout the learning centers. I think more time could have 
been spent on that... I left out of there thinking we still need some work to do in this area. 


E. Conclusions and Recommendations 


In Year Two, CLI offered several strategies to address key implementation challenges identified in 
Year 1, and most lead teachers reported high fidelity to the key elements of the Blueprint approach 
in the classroom: Message Time Plus, Intentional Read Aloud, and Power of Three. 


Yet, teachers were mixed in their perceptions of how easy it is to adapt Blueprint to meet the needs 
of students and implement alongside the Creative Curriculum, in-classroom coaching continues to 
be achallenge, and teacher turnover and variable attendance presented a challenge for service 
delivery. As discussed in the section “Spotlight on Teacher Turnover,” high teacher turnover is 
typical in early childhood education environments. Thus, this project reflects the realities of this 
real-world context. We present the following recommendations to further strengthen the 
implementation of Blueprint in pre-K. 


Recommendations: Blueprint Training and Coaching 


e Consider providing short trainings at centers. Training attendance continued to bea 
challenge in Year Two, and teachers reported that it is difficult to attend trainings on 
Saturdays. CLI could strategize about how to make trainings more accessible to 
teachers. Some teachers and directors suggested that CLI could provide shorter 
trainings at centers. Some centers have occasional professional development or prep 
time when teachers could be available for trainings. 


e Continue to design and implement strategies for effective in-the-classroom 
conferences. It continues to be logistically challenging for teachers to hold coaching 
conferences outside of their classrooms. In Year Two, CLI provided training for coaches 
to conference more effectively with teachers in the classroom. CLI can continue to build 
strategies that support effective in-the-classroom conferences, including leveraging 
assistants and other center staff and planning activities that will keep children actively 
engaged in learning while a teacher is conferencing with their coach. 


Recommended Strategies to Support Teachers in High-Turnover Contexts 


e Train and coach all assistant teachers. If both lead and assistant teachers receive 
Blueprint training and coaching, classrooms are more likely to have more Blueprint 
continuity in the face of turnover. This strategy has been used at some centers, but not 
systematically. 
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e Consider offering each of the three core trainings each year. In Year Two, CLI did 
not offer the Message Time Plus training. As a result, only 35% of lead teachers 
participated. We expect a high percentage of teachers will be new to the Blueprint 
model each year. Thus, we recommend offering each of the three core trainings each 
year to maximize the number of trained lead teachers in Blueprint classrooms. 


e Leverage the knowledge of teachers experienced with Blueprint. CLI staff, coaches, 
and center directors could task teachers who are experienced and confident in 
Blueprint implementation with helping to bring new Blueprint teachers up to speed. CLI 
should still orient new teachers to Blueprint and coach them; however, to provide 
additional support, experienced Blueprint teachers could be entrusted with helping new 
teachers implement core Blueprint practices. 


Recommendations for Differentiating Blueprint 


e Determine if strategies for differentiation should be systematically introduced to 
teachers. Systematically introducing differentiation strategies could ensure that all 
coaches and teachers are on the same page about options for adapting Blueprint. 
Additionally, teachers would have a bank of strategies that might make it easier to plan 
lessons that will realistically and effectively meet the needs of the children in their 
classrooms. However, CLI might determine that this approach does not suit the 
Blueprint model. 


e Expand Blueprint resources that support differentiation. Teachers reported 
utilizing some Blueprint differentiation resources and wanting more. CLI could expand 
these resources by including more Message Time Plus scaffolding suggestions, English 
Language Learner instructional tips, and books in children’s home languages and with 
more diverse representation. 


Recommendations for Using Blueprint with Creative Curriculum 


e Send a clear message. Ensure that CLI staff and coaches send a clear message about 
implementing Blueprint with Creative Curriculum, including clear expectations for what 
pieces of Blueprint are essential to implement. 


e Increase planning time during trainings. In trainings, provide more time and support 
for directors and teachers to plan how they will implement both curricula. To better 
support teachers to implement Blueprint’s most essential elements daily while also 
implementing Creative Curriculum, CLI should provide more time and support for 
directors and teachers to practice making plans that satisfy implementation 
expectations. 


Study 2: Impact of Blueprint on Teachers and Students 


This study focused on the impact of Blueprint professional development and curriculum on teacher 
knowledge and implementation of best practices for early literacy instruction and student 
engagement and skills. Study 2 is divided into three sections: 


e Data and Methodological Approach. In this section, we discuss guiding research 
questions, data collection activities, and comparison center selection criteria for the impact 
study. 


e Impact of Blueprint on Teachers. This section examines the impact of Blueprint on the 
classroom language and literacy environment. In addition, this section examines teacher 
perceptions of how Blueprint is affecting teacher practice. 


e Impact of Blueprint on Students. This section examines the impact of Blueprint on pre-K 


student vocabulary gains. This section also examines teacher perceptions of how Blueprint 
is affecting student engagement and learning. 


A. Data and Methodological Approach 


Research Questions 


To understand Blueprint’s impact in Philadelphia pre-K centers, RFA conducted a mixed-methods, 
quasi-experimental study. The research questions guiding this impact study include: 


1. What is the growth in teacher knowledge and use of instructional practices in pre-K 
centers with the CLI Blueprint program? 


2. What is the growth in pre-K student literacy skills in classrooms with the CLI 
Blueprint program? 


Data and Methodological Approach 


The research activities for this study included the following data collection activities in Fall 2017 
and Spring 2018 (see Methodological Appendix for more information): 


e Beginning- and end-of-year surveys of lead and assistant pre-K teachers (N=60); 


e Interviews with CLI staff and coaches as well as directors and lead teachers at four 
treatment centers; 


e End-of-year researcher observations of classroom language and literacy environments 
using the Early Language and Literacy Classroom Observation pre-K tool (N=73); and 
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e Beginning- and end-of-year direct assessments of student literacy development using the 
Peabody Picture Vocabulary Test, a measure of receptive vocabulary (N=808).11 


To interpret outcome differences in Blueprint and comparison sites as causal evidence of impact, 
we Selected comparison centers that were statistically similar, on average, with respect to key 
factors related to early literacy instruction and outcomes. Our approach was to recruit four more 
sites than needed, compare sites on key factors, and choose a subsample of 11 comparison centers 
similar on average to Blueprint centers (see Methodological Appendix for more information). 


B. Impact of Blueprint on Teachers 


CLI expects that Blueprint will help teachers become more intentional teachers of literacy, 
knowledgeable about both content and pedagogy, with enhanced skills in classroom management 
and the creation of a literacy rich environment. Below, we discuss 1) teacher knowledge and ability 
to implement best practices for early literacy instruction; 2) teacher instructional practice and 
positive classroom culture; and 3) teacher job stress and satisfaction. 


Key Findings: Impacts on Teachers 


Relative to teachers in comparison centers, more Blueprint teachers reported high 
levels of knowledge and ability to implement effective instructional practices for 
teaching language and literacy. 


Blueprint teachers reported that CLI played a significant role in increasing their 
knowledge of best practices in early literacy instruction, including the importance of 
setting high expectations for students and ensuring instruction is intentional and 
engaging 


Observations suggest that Blueprint classrooms have stronger literacy and language 
and literacy environments than classrooms in comparison centers. 


Teacher Knowledge and Ability to Implement Best Practices 


In Spring 2018, we surveyed lead teachers about their perceived knowledge and ability to 
implement effective instructional practices for early literacy and how much they felt their 
knowledge grew from Fall to Spring. To assess the impact of Blueprint on teacher knowledge, we 
compared responses from teachers in centers receiving Blueprint supports to comparison centers. 
Then, we asked Blueprint teachers if they felt CLI supports contributed to their improved 
understanding of best practices. 


11 Receptive vocabulary refers to the words that someone can comprehend. It is a key feature of early language acquisition and is strongly 
correlated with later literacy. Lonigan, Christopher J., and Timothy Shanahan. "Developing Early Literacy: Report of the National Early 
Literacy Panel. Executive Summary. A Scientific Synthesis of Early Literacy Development and Implications for Intervention." National 
Institute for Literacy (2009). 
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Blueprint teachers reported stronger ability to implement best practices, relative to 
teachers in comparison centers. 


As shown in Figure 8, the survey data suggest that Blueprint has positive impacts on teacher ability 
to implement effective best practices in the classroom, though less evidence of impact on teacher 
self-reported knowledge of best practices. 


Figure 8. Percent of lead teachers in Blueprint and comparison centers reporting strong knowledge and 
ability to implement effective best practices in Spring 2018, N=60 teachers 
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Source: RFA teacher survey, 2017-18 
Note: * indicates statistical significance at p<0.05 level from a two-sample test of proportions. 


Specifically: 


e Nearly all (85%) of Blueprint teachers reported high levels of knowledge about effective 
practices, compared to 73% of teachers in comparison centers, though this difference was 
not statistically significant (p<0.244). 


e Nearly all (96%) of Blueprint teachers reported feeling able to implement best practices, 


compared to 73% of teachers in comparison centers, a statistically significant 
difference(p<0.015). 
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Relative to teachers in comparison centers, more Blueprint teachers reported 
considerable growth in knowledge and ability to implement best practices from Fall 


to Spring. 


As seen in Figure 9, more Blueprint teachers reported considerable improvements in knowledge 
and ability to implement effective best practices in early literacy instruction. 


Figure 9. Percent of lead teachers in Blueprint and comparison centers reporting considerable growth in 
knowledge and ability to implement effective best practices from Fall to Spring, N=60 teachers 
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Note: * indicates statistical significance at p<0.05 level from a two-sample test of proportions. 


Specifically: 


e Most (70%) of Blueprint teachers reported considerable growth in knowledge of best 
instructional practices, compared to less than half (45%) of teachers in comparison centers. 
Though this difference was not statically significant (p<0.053), the size of the difference 
between groups is substantively meaningful. 


e Almost twice as many (81%) Blueprint teachers as comparison teachers (44%) reported 
considerable growth in their ability to implement best practices from Fall to Spring, a 
statistically significant difference (p<0.003). 
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Blueprint training, coaches, and resources contributed to teacher growth in 
knowledge and implementation of instructional best practices for early literacy. 


Nearly all surveyed lead teachers reported that CLI contributed to improvements in teacher 
knowledge and ability to implement best practices. In interviews, we asked some Blueprint 
teachers to reflect on what specifically they were learning from CLI supports. Teachers reported 
learning: 


e Foundational understanding is key. Blueprint teachers reported developing a better 
understanding of the foundations of early literacy through a combination of Blueprint 
training, coaches, and resources. 


e To set high expectations for students. One teacher reported learning that setting high 
expectations contributes to strengthening children’s confidence. Another teacher reported 
that, before using Blueprint, instruction was mostly focused on foundational literacy skills. 
Since introducing Message Time Plus, this teacher saw that they were ready to dive deeper 
into literacy and meaning-making. The teacher explained, “Before we started taking the CLI 
training, it was more like, you know, phonics, phonemic awareness, sounds, and rhyming 
words and things like that were more important than sentences, how to build a sentence, 
sentence structure. You would think that'll come more in the [later] grades, but seeing them 
utilize it [during Message Time Plus]—and the confidence it gives them when they do learn 
that—that's really important as well.” 


e To make instruction more intentional and adapt it, when necessary. A teacher said, 
“(Blueprint] really helped me dive in to my goals and objective...If you prepare before the kids 
get there, then you'll have a more rich experience.” Another teacher described learning how 
to be a more intentional teacher through Blueprint training and coaching: “I know what I'm 
teaching. I know what I need to work on. I know my goals. | know my outcomes.” This teacher 
explained that she learned to adapt her lessons extemporaneously because she became so 
aware of her specific, intentional instructional goals. 


e To make instruction engaging. Blueprint taught a teacher that “if you're not excited, [your 
students] won't be excited about something.” This teacher credited Blueprint coaching with 
adding tools like using movement to teach “snap” (high-frequency) words to make instruction 
more engaging. Another teacher reported learning “how to get the kids really interested in the 
books and stuff using my personality while reading the story.” 


Underlying these improvements may be an increase in teacher confidence, as our interviews 
suggested. One CLI staff person and one center director agreed that they saw teacher confidence 
improving throughout the intervention. A CLI staff person said: 


I've seen growth in confidence. And not just with Blueprint—I think I see a growth in 
overall how [teachers] feel about themselves as educators. They feel a lot better about 
their teaching because they see that their students are growing. So when they see the 
growth of the students, we get [teachers] to know, that's directly because of you. 
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A teacher confirmed this staff member’s theory, explaining, “It just builds your confidence to come 
in. After you finish implementing it and you see the results, it's like, ‘I have more confidence in 
doing this. I want to keep doing this.’ I feel like that's what's really good about it.” 


Teacher Job Stress and Satisfaction 


In addition to supporting teacher knowledge and ability to implement best practices, CLI aims to 
improve working conditions for teachers by investing in their skills and supporting them as 
professionals. We drew from surveys and interviews with teachers to understand the impact of CLI 
supports on teacher job stress and satisfaction. 


Relative to teachers in comparison centers, fewer Blueprint teachers reported feeling 
under great stress at work, and more reported feeling very satisfied with their job. 


Survey results suggest that Blueprint positively impacts teacher work environments, reducing 
stress and increasing job satisfaction. Figure 10 shows the results of our survey asking teachers 
about the stress feel at work. Relative to comparison teachers, fewer Blueprint teachers (15% vs 
45%, p=0.023) report feeling under great stress “often” and more (37% vs 12%, p=0.011) report 
“never or rarely.” 


Figure 10. Percent of lead teachers in Blueprint and comparison centers that report feeling under great 
stress in the past month, N=60 teachers 
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We also asked teachers about their job satisfaction. As Figure 11 demonstrates, about two-thirds of 
Blueprint lead teachers reported feeling “very satisfied” with their job, while just over a third of 
comparison lead teachers reported the same. This difference, however, is not statistically 
significant. 


Figure 11. Percent of lead teachers in Blueprint and comparison centers that report satisfaction with their 
job in the past month, N=55 teachers 
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Note: * indicates statistical significance at p<0.05 level from a two-sample test of proportions. Five teachers reporting “Neither satisfied 
nor dissatisfied” were excluded from this analysis. 


All surveyed Blueprint teachers who reported feeling either “very” or “somewhat” satisfied with 
their job reported that working with CLI improved their job satisfaction. Our interviews suggested 
specific aspects of Blueprint participation affect job satisfaction in a variety of ways. For example, 
one teacher reported that seeing children learn and grow supports job satisfaction. Another teacher 
explained that Blueprint’s intentional instruction can help manage behavior, making their job 
easier: “Sometimes, with behaviors that we have, sometimes it's a challenge, but with Blueprint, it 
gives you a guide on how to implement the lesson to them so that they'll learn and behave.” 


Yet our interviews with CLI coaches suggested that there are limits to what CLI, or any professional 
development intervention, can do for pre-K teachers. For example, when asked whether working 
with CLI to implement Blueprint has an effect on pre-K teachers’ job satisfaction, one CLI coach 
explained that it does not because the high demands of the profession, including being “inundated 
with so much paperwork,” reach beyond the scope of the Blueprint intervention. 
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Classroom Language and Literacy Environment 


Blueprint also supports teachers with specific strategies to make their classrooms more literacy- 
rich. To understand impact on classroom language and literacy environment, RFA observed 73 
Blueprint and comparison classrooms using the Early Language and Literacy Classroom 
Observation, pre-K tool (ELLCO). More about the observation process and inter-rater reliability of 
RFA’s observations can be found in Appendix A. 


The ELLCO provides data on two aspects of the classroom environment: 


e The general classroom environment subscale consists of two components: 
classroom structure and curriculum. Classroom structure addresses classroom 
IN organization and contents, children’s access to and use of materials, management 
oy practices, and adult roles and professional focus. Curriculum consists of the 
curriculum environment, instructional strategies, opportunities for child choice and 
initiative, and recognition of diversity in the classroom. 


e The language and literacy environment subscale consists of three components: 
language environment, books and book reading, and print and early writing. 
Language environment addresses discourse climate in the classroom, opportunities 
for extended conversations, vocabulary development, and efforts to develop 
phonological awareness. Books and book reading addresses the organization and use 
of the book area, the characteristics of books available, the presence and use of 
books across content areas of the curriculum, and the quality and frequency of book 
reading. Finally, print and early writing includes the availability of writing materials, 
opportunities that build awareness of print and varied purposes of writing, 
instructional strategies, and use of environmental print. 


To analyze the impact of Blueprint on classroom environment, we conducted a statistical test (t- 
test) of the difference in average scores in Blueprint classrooms compared to comparison 
classrooms. We present average differences but also, to help understand the magnitude of impact, 
present effect sizes. An effect size takes the difference in averages across groups relative to the 
known population variation in the scale (the population standard deviation). An effect size of .8 is 
generally considered large, .5 is medium, and .2 is small.12 


CLI supports moderately improved the language and literacy environment, but there 
is continued room for improvement. 


Below, we present the average scores for Blueprint and comparison classrooms and describe 
differences in terms of effect sizes. As Figure 12 shows, CLI supports had a moderate positive effect 
on the language and literacy environment of study classrooms (effect size=0.50, p=0.03). The 
average score of 2.9 out of 4.0 for CLI-classrooms suggest some room for continued improvement. 
CLI-supported classrooms and non-CLI supported classrooms scored high on the more general 
measure of classroom environment. 


12 Cohen J. (1988). Statistical Power Analysis for the Behavioral Sciences. New York, NY: Routledge Academic 
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Figure 12. Classroom environments of Blueprint and comparison classrooms, N=73 classrooms 
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Note: * indicates statistical significance at p<0.05 level from a t-test of mean differences. 


C. Impact of Blueprint on Students 


CLI expects that by helping teachers increase their knowledge and skills in early language and 
literacy instruction, students will in turn become more engaged in language and literacy learning 
and gain language and literacy skills. Below, we discuss CLI impact on 1) student literacy skills and 
2) student engagement. 


Key Findings: Impact on Students 


Relative to children in comparison centers, children in Blueprint centers have 
stronger growth in literacy skills, according to direct assessments. 


Interviews revealed additional potential benefits of Blueprint, such as increased love 
of reading and writing among children, increased self-esteem among children, and 
parental engagement in literacy practices. Teachers cited the key elements of 
Blueprint as beneficial. 
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Student Early Literacy Skills 


We conducted multiple research activities to understand the impact of Blueprint on children’s 
learning and literacy skill development, including directly assessing children’s vocabulary growth, 
surveying teachers about their perceptions of children’s literacy skills, and interviewing Blueprint 


teachers about how Blueprint improved children’s skills. 


Figure 13. Receptive vocabulary among students in 
Blueprint and comparison classrooms, predicted Fall 
and Spring growth scale scores, N=808 students 
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Note: * indicates statistical significance at p<0.05 level from a multi- 


level regression model (see Appendix for more information). 


Children in Blueprint classrooms 
outpaced their peers in literacy skill 
development. 


PPVT data show that the difference in 
average spring PPVT growth scores is 3.6 
points higher for the treatment group, 
with a statically significant effect size of 
22% of a standard deviation (Figure 13). 
In terms of months of growth, children in 
CLI-served classrooms made 2-3 months 
of additional progress in vocabulary 
development compared to children in 
similar classrooms not supported by CLI. 


While receptive vocabulary is a key 
marker of early literacy, we surveyed 
teachers to understand impacts on 
language and literacy skills more 
generally. Teacher perceptions of growth 
mirror the PPVT findings. We see stronger 
outcomes in Blueprint classrooms than in 
comparison classrooms, as shown below 
in Figure 14, though the comparison 
between Blueprint and comparison 
teachers is not statistically significant. 
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Figure 14. Percent of lead teachers in Blueprint and comparison centers reporting progress in student 
literacy skills from Fall to Spring, N=60 teachers 
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Specifically: 
e Nearly all (96%) of surveyed Blueprint teachers reported their students had made “quite a 


bit” or “a great deal of progress” in language and literacy activities since Fall 2017, 
compared to 79% of comparison teachers (0.047). 


e Nearly all (97%) of Blueprint teachers attributed student progress to supports from CLI. 
Teachers also reported in interviews that students in Blueprint classrooms were learning: 
e Foundational skills. “They're learning letters, letter sounds, letters and words.” 
e High-frequency or “snap” words. “They're learning, like, cheers they can use, they're 
learning snap words and different ways to memorize those words. It's not just, ‘Okay, this is 
on the word wall.’ ‘G-O. Go! What is it? G-O. Go! We’re going to do a cheer to go with go!’ So it 


kind of sticks with them a little bit more.” 


e Howto follow a story. “They're learning story progression, they're learning content, 
they're learning how to read a book.” 
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e Howto use a physical book. “She sat on the floor, she sat beside me, and she was just 
talking to me, moving her lips. And she was pointing, and she was holding the book the 
proper way. And she was looking at it the proper way and she was turning the pictures and 
she actually went back and looked at different pictures. And she's only three.” 


e Writing. “They're learning how to write a sentence, how a writer writes, how a writer 
brainstorms and comes up with the ideas.” 


Student Engagement and Positive Classroom Culture 


Blueprint uses the Power of Three to help teachers foster positive classroom cultures. The clear 
objectives in Blueprint lessons also aim to support student engagement. In this section, we report 
survey results and interviewees’ perceptions of how Blueprint affects classroom culture and 
student engagement. 


Children in Blueprint classrooms showed increases in engagement from Fall to Spring 
in language and literacy activities, compared to children in comparison classrooms. 


As seen in Figure 15, Blueprint teachers perceived more improvement in student engagement from 
Fall to Spring relative to comparison teachers. However, the proportion of teachers reporting that 
students were either “very or “extremely” engaged was similar across both groups. 


Figure 15. Percent of lead teachers in Blueprint and comparison centers reporting levels and growth in 
student engagement from Fall to Spring, N=60 teachers 
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Specifically: 


Most Blueprint teachers (89%) and comparison teachers (82%) reported high levels of 
student engagement in language and literacy activities. 


Nearly all (93%) of Blueprint teachers reported significant growth in student engagement 
from Fall to Spring, compared to 76% of teachers in comparison centers (p<0.047). 


As shown above, nearly all (87%) of Blueprint teachers reported that CLI contributed to increases 
in student engagement. In interviews, teachers said that children are generally more confident and 
engaged in literacy activities as a result of Blueprint. One teacher noted that children are engaged in 
learning throughout the day: “It's showing them that learning is fun. It's not...like, ‘Oh, reading 
again? Ugh. Writing again? Ugh.’ It's so much different things going on in that one day. It amazed me 
how much stuff goes on in one day and how much they get in one day.” 


Specific elements of Blueprint support children’s engagement, including Message 
Time Plus, Intentional Read Aloud, and Turn-and-Talk. 


In interviews, teachers, site leaders, and coaches identified specific elements of Blueprint that 
support children’s engagement: 


Message Time Plus. As in Year One, teachers especially noted children’s strong 
engagement with Message Time Plus. One center director explained, “Message Time Plus is 
really beneficial. Just the structure involving the students and getting student voices is 
really beneficial, because it really helps build confidence and helps the students to get 
involved.” 


Intentional Read Aloud. Some coaches and teachers said that children are more engaged 
in stories. One coach explained, “They just seem so excited about the books, because we 
read [the same] books sometimes two or three times in a week.” One teacher reported that 
children in her classroom act as the teacher during center time and reenact read-alouds. 
The teacher said, “I'm just listening and she's like, ‘Well, this is the front cover of the book, 
and this is the back cover, and this is the title page. Don't you like the title page? Look how 
colorful it is.’ ... I'd say they're more engaged with the reading.” 


Turn-and-talk. Two teachers reported that using “turn-and-talk” instructional practices 
helps to engage children. One explained, “Because we do turn-and-talk, to have like little 
partners sometimes, it keeps them actively engaged. “ 


Teachers attributed some positive changes in behavior to Blueprint’'s Power of Three. 


In interviews, a few teachers and one coach reported that children are becoming better at 
expressing themselves and their feelings because the Power of Three provides students with 
positive vocabulary. One teacher explained, “You hear them actually making themselves 
responsible for their own actions. And for three- and four- and five-year-olds, that's pretty good.” 


34 


Conclusions and Recommendations 


In this section, we review key findings from the implementation and impact studies and put forth 
suggestions for research in Year Three, the final year of the intervention. 


CLI used several strategies to improve implementation challenges in Year Two, yet 
more support may be needed. 


In Year Two, CLI employed various strategies to address implementation challenges identified in 
our earlier report. For example, CLI developed content and provided trainings for coaches and 
teachers to support productive in-the-classroom conferences, the implementation of Blueprint and 
Creative Curriculum, and differentiated instruction. While most teachers felt that differentiating 
Blueprint and using Blueprint alongside Creative Curriculum was nota challenge, some teachers 
continued to struggle and need additional support. 


Providing professional development in the context of high teacher turnover is a 
significant challenge. 


By the end of Year One, 37 lead teachers were working in the 11 study centers receiving CLI 
support. By the end of Year Two, only half of those teachers were still working in study centers. The 
rate of teacher attrition was similar in comparison classrooms. 


This similar rate of attrition is notable considering that, relative to teachers in comparison centers, 
fewer Blueprint teachers reported feeling under great stress at work and more reported feeling 
very satisfied with their job. In our interviews, teachers conveyed that living wages and benefits 
greatly contribute to teacher retention, and that professional development may not be enough to 
reduce teacher turnover. As one teacher from a site with low turnover explained, a key reason that 
teachers continue to work in or leave pre-K sites is how much they are paid: “No matter how much 
you like your job, you still have to live.” 


We find that providing the full “dose” of CLI training and coaching is a significant challenge ina 
high-turnover environment; fewer than one fourth of lead teachers working in Blueprint 
classrooms in June 2018 had been to all three core Blueprint trainings and received 40 hours of 
coaching. This reflects the reality of providing professional development in a context where high 
teacher attrition is typical, such as in early childhood education. 


While it is out of the scope of CLI’s supports to address the low wages that plague the early 
childhood workforce, we make some recommendations to enable CLI to sustain center-level gains 
in teacher knowledge and ability to implement effective best practices in early literacy: 


e CLI should consider training and coaching assistant teachers alongside lead teachers. 
Following this recommendation would likely have multiple benefits. First, we found that 
supportive assistant teachers strengthen the implementation of Blueprint in the classroom. 
Thus, even in classrooms with low teacher turnover, assistant teachers who are trained and 
coached are more able to support high-quality implementation of key elements of Blueprint. 
Second, several lead teachers in classrooms in Spring 2018 were assistant teachers earlier 
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in the year. Continuity of Blueprint implementation would be easier if assistants were 
trained and coached. 


e CLI should consider offering each of the three core trainings every year, because a high 
percentage of teachers will be new to the Blueprint model. In Year Two, CLI did not offer the 
Message Time Plus training. As a result, only 35% of lead teachers participated in the 
Message Time Plus training. 


Blueprint implementation and impacts were strong despite evidence of limited 
training and coaching due to turnover and variable attendance. 


We found that very few centers and classrooms received the full benefit of training and coaching, 
largely due to the high rate of teacher turnover. Despite this, most teachers reported implementing 
key elements of Blueprint with fidelity, including daily use of Message Time Plus and Intentional 
Read Aloud and integration of the Power of Three into the classroom culture. We also found strong 
evidence of impact on teachers and students, demonstrating across multiple sources of data that 
teachers and children in Blueprint centers were benefiting from CLI supports and the Blueprint 
curriculum. The rigor of the quasi-experimental design, specifically the statistically comparability of 
treatment and control centers, allows us to interpret these findings as causal positive impacts on 
teacher and student outcomes. 


This apparent disconnect between limited intervention fidelity and strong inplementation and 
impact in classrooms suggests that, even in a setting with high turnover, the Blueprint 
intervention can have a strong impact on improving the foundations of literacy for children. 
Our study was not designed to tease apart which aspects of the intervention are most impactful and 
which aspects may be less central for driving improvements. Quantitative data to support this 
analysis would need to involve a larger sample of teachers and students. While increasing the 
number of teachers and students in the Year Three study is not within the scope of this work, we 
suggest including qualitative data to understand teachers’ and directors’ perceptions of the 
components of the intervention that are critical to driving positive and lasting change. 
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Appendix A: Data Collection and Analytic Methods 


A. Teacher Survey Sample Description 


Instrument development. RFA developed a 60-item questionnaire (45-item for comparison 
teachers), with open- and closed-response items for teachers to generate cross-sectional data on 
teacher perceptions of their knowledge and ability to implement best practices in early literacy 
instruction, student learning and engagement, and workplace environment. For Blueprint teachers, 
we also asked teachers to reflect on experiences with Blueprint professional development and 
implementation of the Blueprint curriculum. 


Sampling frame and survey administration. In May 2018, RFA used Qualtrics to program and 
administer a web-based survey to lead teachers who had consented to our study. The survey was 
live for four weeks, and three reminder emails were sent to non-respondents during that time. 
Teachers provided their email addresses when they signed consent forms. 


Response rate. Of the 74 lead teachers that consented to the study, RFA received surveys from 60, 
a response rate of 81%. The response rate was not statistically different across treatment and 
comparison sites (80% vs 85%, respectively, p=0.749). 


The final teacher survey sample size is N=60, or 33 comparison teachers and 27 treatment 
teachers. 


e Centers represented. RFA received at least one teacher survey from 21 of 22 centers. Ten 
of the 11 treatment sites were represented in the teacher survey, and all 11 comparison 
sites were represented. 


e Classrooms represented. The teacher survey represents about 80% of the study 
classrooms; RFA received at least one teacher survey from 58 of the 73 classrooms (25 of 34 
treatment classrooms [74%] and 33 of 39 comparison classrooms [85%]). 


A note on representativeness of lead teacher survey data: 
Our analysis does not account for variation in the number of teachers per site and the number of 
teachers responding per site. This is due to a small overall sampling size. That is, the data cannot 


support strategies to account for the structure of the data, e.g., weighting results based on the 
number of classrooms per center or estimating multi-level models. 


B. Interviews 


Protocol development. RFA developed open-ended interview protocols to understand Year Two 
implementation and perceived impacts of CLI training and coaching on teacher and student 
outcomes. 


Participants. RFA interviewed four directors and eight teachers in four treatment centers. We 
strategically sampled directors and teachers to go deeper into issues that were raised in Year One: 
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e Director support and knowledge of Blueprint. CLI staff reported that two selected sites 
had high director support and knowledge of Blueprint and two selected sites had low 
director support and knowledge of Blueprint. 


e Teacher turnover. Two sites had high teacher turnover; two sites had low teacher 
turnover. 


e Differentiation of Blueprint. Teachers in our sample had both high and low proportions of 
English Language Learners and younger learners. 


e Implementing Blueprint with Creative Curriculum. All four sites balanced using 
Blueprint with Creative Curriculum. 


e Experience with Blueprint. Half of the interviewed teachers were new to implementing 
Blueprint in Year Two; the other half implemented Blueprint in Year One 


e Strength of implementation. Half of the interviewed teachers were identified by CLI 
coaches as strong Blueprint implementers; the other half were identified as weaker 
Blueprint implementers. 


We also conducted interviews with three CLI staff and all four coaches. 


Data analysis. Interviews were recorded (with consent), transcribed, and coded using the Dedoose 
web application. The data were then systematically analyzed to understand Year Two 
implementation and perceived impacts of CLI training and coaching on teacher and student 
outcomes. 


C. Baseline Equivalence 


Comparison Site Selection 


In Year One of RFA’s CLI Blueprint evaluation, CLI and RFA worked together to recruit a total of 26 
STAR 3 and 4 pre-K centers in North, West, and South Philadelphia. Eleven of the centers received 
Blueprint training and coaching January 2017-June 2018, and 15 served as a comparison group. 
Drawing from STAR 3 and 4 centers across the city, we already reduced much of the variation in 
pre-K center quality. Additional key aspects of comparison site selection included: 


e High need. To create a matched comparison group, CLI also selected comparison sites from 
areas with a high need for quality child care. 


e Geography. CLI selected sites in the same geographic areas as the treatment sites. 
e Approval by district. Because several SDP sites participated in the treatment group and 
because SDP provides supports to many centers, CLI submitted the list to SDP for their sign- 


off. 


e Relationship with CLI. CLI sought sites with no previous relationship with CLI. 
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Methodological Strategies to Address Non-random Selection of Centers 


In Year Two, we employed several strategies to address the non-random selection of centers in 
order to estimate impacts of Blueprint on teacher and student outcomes. 


Specific strategies included: 


e Collecting data via teacher surveys on characteristics of centers, directors, and teachers to 
more fully assess differences in the contexts of treatment and comparison centers; 


e Collecting pre-test student outcome data to account for pre-existing differences across 
treatment groups; and 


e Drawing from an over-recruited sample of comparison centers to identify a subset that 
most closely matched the centers in our treatment group. 


We recommended dropping two centers for non-statistical reasons. One center served a high 
proportion of students with special needs, and the other had consistently had poor response rates 
across data collection activities. 


To identify an additional two centers to exclude, we estimated a statistical model predicting 
treatment status based on two variables that are highly correlated with other center 
characteristics: baseline PPVT scores and whether the center was an SDP site. Examining the 
overlap of predicted treatment status across treatment and comparison centers revealed limited 
overlap at the high end of the distribution of propensity scores. This created a disproportionately 
lower average propensity for treatment among comparison centers. To address this, we excluded 
the two comparison sites with the lowest propensity for treatment. This resulted in a more even 
distribution at the lower end to reduce disproportionately low propensity comparison sites. 


Baseline Equivalence of Treatment and Control Centers 
As Table 1A shows, the treatment and comparison sites are statistically equivalent on key 


characteristics relevant to early literacy. Importantly, the pre-test measure of the main outcome of 
interest is evenly distributed across groups. 
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Table 1A. Statistical comparison of center characteristics, t-tests 


Center Characteristic 


Comparison 


(oD) 


Treatment 
(ae sD) 


Number of school district sites 2 2 - - 
Number of pre-K classes 3.54 (1.63) 3.09 (2.12) 0.45 0.579 
Average class size 15.98 (3.06) 15.88 (4.06) 0.10 0.951 
Average prop ELL per class 0.13 (0.24) 0.15 (0.29) -0.01 0.912 
Average prop IEP per class 0.08 (0.05) 0.05 (0.05) 0.03 0.211 
Fe ee eects bee 0.10 (0.04) 0.10 (0.08) <0.01 0.958 
class 

Prop classes - Head Start 0.42 (0.35) 0.33 (0.40) 0.09 0.563 
Prop classes - PreK Counts 0.39 (0.38) 0.23 (0.40) 0.15 0.367 
Standard PPVT score 91.37 (3.93) 93.12 (7.00) -1.75 0.479 


D. Observations of Classrooms 


The Early Literacy and Language Classroom Observation Research Edition Pre-K Tool (ELLCO) 
measures teacher language and literacy instruction and general classroom environments. Each 
ELLCO item is rated with a 5-point anchored scale that provides descriptions of ratings at each 
level. 


ELLCO Data Constructs. The 19 ELLCO observation items are grouped into five constructs: 13 


e The Classroom Structure construct measures the physical environment of the classroom, 
including the layout, traffic flow, an inventory of the materials, as well as how the children 
use them. It also includes observations of classroom management and adult roles within the 
classroom. 


e The Curriculum construct measures the use of time throughout the observation, the 
integration of themes, and the recognition of child diversity and choice in the daily routine. 


e The Language Environment construct measures the discourse climate, how teachers 
interact with the children, and how they build vocabulary and phonological awareness 


through extended conversations. 


e The Books construct measures the organization, characteristics, availability, and use of 
books for learning. It also measures the teacher’s approach to reading a book aloud to 


13 Smith, M. W., Brady, J. P., & Anastasopoulos, L. (2008). User's Guide to the Early Language & Literacy Classroom Observation Pre-K 
Tool. Baltimore: Paul H. Brookes Publishing Co. 
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children and their general engagement during the read-aloud. 


e Finally, the Print and Early Writing construct measures the use of environmental print, 
writing materials, and writing instruction. 


These five constructs are further grouped into two subscales - the General Classroom Environment 
subscale and the Language and Literacy subscale. The General Classroom Environment subscale is a 
composite of the Classroom Structure construct and the Curriculum construct. The Language and 
Literacy subscale is a composite of the Language Environment, Books and Book Reading, and Print 
and Early Writing constructs. Smith et al. (2008) report an internal consistency of 0.86 for the 
Language and Literacy subscale and of 0.83 for the General Classroom Environment subscale. 
ELLCO observational scores are predictive of children’s receptive vocabulary (as measured by the 
Peabody Picture Vocabulary Test) and early literacy scores (as measured by the Profile of Early 
Literacy Development). 


Observation training and inter-rater reliability. To establish inter-rater reliability, RFA 
observers received ELLCO training and practiced observing classrooms in March-May 2017. 
Practice sessions and debriefing conversations were used to calibrate ratings after each 
observation to achieve inter-rater reliability (i.e., agreement within one point on each item). 


E. Student Outcomes: Direct Assessment 


In Year Two, RFA collected student vocabulary assessment data by administering the PPVT to 
students enrolled in study centers. Our sample included children who were proficient in English" 
and whose caretakers did not opt them out of participating. RFA staff and external consultants, 
hired by RFA, collected baseline data from October-December 2017 and post data from March-June 
2018. Assessment data were considered valid if students passed the PPVT training items, had a raw 
score of at least 4, and completed the assessment.15 


e Fall 2017 sample. Of the 1,242 students enrolled in study centers in Fall 2017, RFA 
collected valid assessment data from 1,092 students, a response rate of 88%. The sample 
was comprised of 458 students from treatment centers and 634 students from comparison 
centers. 


e Spring 2018 sample. Of the 1,121 students enrolled in study centers in Spring 2018, RFA 
collected valid assessment data from 1,063 students, a response rate of 95%. In the spring, 
487 students were from treatment centers and 576 students were from comparison 
centers. 


14 We initially collected baseline data for students proficient in Spanish using the Spanish-language version of the PPVT, the TVIP. We 
discontinued use of the TVIP during spring data collection because we determined the Spanish-language version was not comparable to 
the English version based on the following reasons: the TVIP is in black and white and less engaging for young children; the scoring 
process is different for the TVIP; the TVIP does not include the Growth Scale Value (GSV); and there are fewer total items in the TVIP, 
providing an advantage to earn a higher score to students taking the PPVT. In practice, we did not find these assessments to be parallel 
and ultimately excluded TVIP data from the sample. 

15 Citation: Dunn, L. M., Dunn, D. M., Lenhard, A., Lenhard, W., & Suggate, S. (2015). PPVT-4: Peabody picture vocabulary test; [manual]. 
Pearson. 
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e Final Year Two sample. Students for whom RFA had valid data from Fall and Spring were 
included in the analysis (N=808, 70% of the full sample). There were not differential 
response rates across treatment and comparison groups (70% vs 69%). The final sample 
includes data from 32/34 treatment classrooms and 39/40 comparison classrooms. 


Dependent variable. The student outcomes measure was the Growth Scale Value (GSV) score from 
the PPVT assessment. The GSV was used because it can measure change in scores over time ona 
single, continuous scale, allowing us to compare scores over time and to determine a program’s 
effectiveness. 16 


Multi-level model. RFA estimated a multi-level random intercept model, adjusting for the 
following student-level characteristics: baseline PPVT growth scores, ELL status, and age. Multi- 
level modeling provided an estimate of the difference in average GSV scores between treatment and 
comparison groups, accounting for clustering of students within centers and student-level controls. 


Student-level model (Level 1). We began by modeling a child’s GSV from spring PPVT as a 
function of select student-level characteristics (i.e., baseline scores, child’s age, ELL status) and a 
student-level random error (i.e., e; ): 


PPVTSpring;; = Boj + 6, ;PPVT Fall; + f2;ELLStatus; + B3j;Agei + ej 


Bo; is the mean GSV score of center j, after accounting for differences in students’ baseline scores, 
age, and ELL status. 


e; is the difference between an individual student’s PPVT score and his/her/their center’s mean. 


Center-level model (Level 2). At the Level 2, we specified the adjusted center-level means area 
function of the treatment condition. However, slope coefficients 6,, 82,and 63 are assumed to be 
fixed at level 2. We examined an alternative specification that allowed these slope coefficients to 
vary with a random error at the center-level. This alternative model was rejected because it did not 
improve overall model fit compared to the model presented here. 


Boj = Yoo + YoiTreatment + uo; 


Bij = Y10 
B2j = Y20 
B3; = Y30 


Yoo represents the grand mean of comparison centers, where treatment = 0 (i.e., comparison 
centers.) 


Yo1 represents the treatment effect, which is the difference between the treatment mean and 
comparison mean, adjusting for student-level covariates. 


16 Dunn, L. M., Dunn, D. M., Lenhard, A., Lenhard, W., & Suggate, S. (2015). PPVT-4: Peabody picture vocabulary test; [manual]. Pearson. 
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Y10 ¥20, and Y29 represent the effects of PPVTFall, ELLStatus, and Age on the dependent variable 


(PPVTSpring), respectively. 


Uo; represents the center-level error term, i.e., the difference between a center’s mean and the 


grand mean of all students. 


Mixed model (Levels 1 and 2 combined). 


PPVTSpringij = Yoo + YoiTreatment + 7¥i9PPVT Fall; + Y2oELLStatus; + y39Agei + ej; + Uoj 


Table 2A. Mixed-model coefficients 


Covariates Coefficient elitelleoM-lacele P-value 
Treatment effects (center level) 3.60** 1.30 0.005 
Baseline (PPVT Fall GSV, student level) 0.62*** 0.02 0.000 
ELL status (student level) -1.76 1.64 0.001 
Age (student level) -4.46*** 1.36 0.284 
Intercept (adjusted grand mean) 49.35*** 2.63 0.000 


*p<0.05; **p<0.01; ***p<0.001 


Effect size. RFA calculated effect size as the difference between treatment and comparison groups’ 
mean GSV scores divided by the standard deviation of the normed population. The standard 
deviation of the GSV of the normed population sample varies by age. Because most students in this 


study were 5-5 ¥% years old at the time of spring assessments, RFA calculated the effect size of 
Blueprint based on the standard deviation of the normed population for this age group (SD=16.5). 
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